Online social interaction, education, and health care by analysing affect and cognitive features

ABSTRACT

A method of establishing a collaborative platform comprising performing a collaborative interactive session for a plurality of members, and analyzing affect and/or cognitive features of some or all of the plurality of members, wherein some or all of the plurality of members from different human interaction platforms interact via the collaborative platform, wherein the affect comprises an experience of feeling or emotion, and wherein the cognitive features comprise features in a cognitive state, the cognitive state comprising a state of an internal mental process.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claim to priority to the U.S. Provisional ApplicationNo. 61/720,405, entitled “The Next Generation of Virtual Live Education”filed on Oct. 31, 2012 and U.S. Provisional Application No. 61/719,980,entitled “Online Social Interaction, Education, and Health Care byAnalysing Affect and Cognitive Features” filed Oct. 30, 2012; U.S. Ser.No. 13/455,133, entitled “VIRTUAL COMMUNICATION PLATFORM” filed Apr. 25,2012, which claims benefit from U.S. Provisional application Ser. No.61/556,205, entitled “SWAP: FUTURE OF VIDEO CHATTING,” filed on Nov. 5,2011, and U.S. Provisional application Ser. No. 61/625,949, entitled“SWAP The Next Generation of Virtual Communication Platform,” filed onApr. 18, 2012, and all of the above applications are incorporated hereinin their entirety by reference. All U.S. patents and publications listedin this application are incorporated herein in their entirety byreference. This application is also related to the U.S. patents andpublications listed in Appendix 1. These U.S. patents and publicationslisted in Appendix 1 are incorporated herein in their entirety byreference.

BACKGROUND

According to International Data Corporation (IDC), a global provider ofmarket intelligence, video communications is one of the most promisingindustries with the potential to create a market of at least 150 millionpeople in America alone in the next five years.

Certain video communication platforms for groups of individuals tocreate and share information, interact with each other through thesoftware and generally use the software to achieve an individual orgroup objective are currently available. Generally these systems storethe collaboration for future reference and further discussion orcollaboration. However, these systems have several limitations that havebeen addressed herein. Also, novel solutions for these limitations areprovided herein.

SUMMARY

The embodiments herein relate to a method of establishing acollaborative platform comprising performing a collaborative interactivesession for a plurality of members, and analysing affect and cognitivefeatures of some or all of the plurality of members.

In one embodiment, some or all of the plurality of members fromdifferent human interaction platforms interact via the collaborativeplatform,

One embodiment can further comprise displaying of targetedadvertisements or notifications based on the context of the interactivecollaborative session.

One embodiment can further comprise measuring effectiveness of thedisplaying of targeted advertisements or notifications.

One embodiment can further comprise integrating an application or adevice within the collaborative interactive session.

Another embodiment relates to a computer implemented system comprising:a storage medium configured to store a collaborative interactive sessiondata; and a processor configured to perform a collaborative interactivesession for a plurality of members, wherein the system analyses affectand cognitive features of some or all of the plurality of members.

In one embodiment, some or all of the plurality of members fromdifferent human interaction platforms interact via the collaborativeinteractive session, wherein the different human interactions platformscomprise social media platforms.

In one embodiment, the system is further configured to display targetedadvertisements or notifications based on the context of the interactivecollaborative sessions.

In one embodiment, the system is further configured to measureeffectiveness of the displaying of targeted advertisements ornotifications.

In one embodiment, the system is further configured to integrate anapplication or a device within the collaborative interactive session.

In one embodiment, the system comprises a sound and/or video hub,wherein the sound and/or video hub allows any member of the plurality ofthe members to play a song and/or a video and simultaneously allows someor all of the plurality of members to listen and/or watch the songand/or the video played.

In one embodiment, the system comprises audio and/or video synopsis ofthe collaborative interactive session for the plurality of members usinga sound and image-processing technology that creates a summary of anoriginal full length audio and/or video.

Another embodiment relates to a tangible non-transitory computerreadable medium comprising computer executable instructions executableby one or more processors for establishing a collaborative platformcomprising performing a collaborative interactive session for aplurality of members, and analyzing affect and cognitive features ofsome or all of the plurality of members.

In one embodiment, some or all of the plurality of members interact fromdifferent human interaction platforms.

One embodiment could further comprise computer executable instructionsexecutable by one or more processors for displaying of targetedadvertisements or notifications based on the context of the interactivecollaborative sessions.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featurescan become apparent by reference to the drawings and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an embodiment of data passing through the SWAP platform,wherein data is acquired and the multimedia segmented and analysed.

FIG. 2 shows an embodiment of chatting threads.

FIG. 3 shows an embodiment of profile appearance.

FIG. 4 shows an embodiment of the analysis of data through SWAP for‘close friends’.

FIG. 5 shows an embodiment of e-learning showing an user interface forstudying from a video lecture.

FIG. 6 shows an embodiment of a virtual classroom with e-material usedas learning medium, based on study of eye movement, pupillary dilationand facial study.

FIG. 7 shows an embodiment of a user interface.

FIG. 8 shows an embodiment of a collated version of the user interface.

FIG. 9 shows an embodiment of how insurance or health care companieswill acquire date through the cell phone of a client.

FIG. 10 shows an embodiment of how the media acquired by SWAP is goingto be analyzed.

FIG. 11 shows an embodiment of a non-invasive patient tracking method.

FIG. 12 shows a flow diagram that delineates possible trackingmechanisms.

DETAILED DESCRIPTION

Large amount of online media that is transferred is merged providingconvenience to user. This data is analysed to find out affect andcognitive state. Utilising this data a new form of social interactionplatform is developed which will incorporate many features of real humaninteraction.

The term “affect” refers to the experience of feeling or emotion. Affectis a part of the process of an organism's interaction with stimuli. Theword also includes affecting display, which is a facial, vocal, orgestural behavior that serves as an indicator of affect.

The term “cognitive state” refers to the state of exploring internalmental processes, for example, to study how people perceive, remember,think, speak, and solve problems.

SWAP is the acronym of an embodiment of a virtual communication platformsystem described herein. SWAP and a virtual communication platformsystem are used synonymously in this application.

Embodiments herein relate to SWAP, which can be a web-based applicationthat serves as a multi-dimensional platform for peer-to-peercommunication. Current video communication services such as Skype onlyprovide basic face-to-face contact pathways—the interaction is limitedto text, audio, and video. SWAP integrates collaboration withcommunication. It streamlines the base services of peer-to-peer text,audio and video communication with interaction on various collaborativeplatforms as well as with individual web-based activity. SWAP canincorporate existing streams of social media.

SWAP strives to be the global leader in providing a unifiedcollaboration platform using Internet communication media whileenhancing the capabilities of virtual interaction of people from allwalks of life. SWAP can provide young adults with a video communicationsapplication that integrates multiple streams of online media withvirtual interaction. SWAP can provide a unified platform that allowsusers of any social media service, such as FACEBOOK® or GOOGLE+®, tointeract on, removing the fragmentation within social mediacommunication. This platform also combines text, audio, and videocommunication with collaboration in the areas of academia, music, andrecreational activities such as gaming, extending the capabilities ofcurrent virtual communication.

This application can be organized into several spheres of interactionknown as “globes”. Each globe can provide a base interaction formultiple users to collaborate. Our application can integrate thesecollaboration platforms with a video feed to enhance overall virtualinteraction.

The data passing through the SWAP platform will be acquired and themultimedia will be segmented and analysed. This can be seen in FIG. 1.FIG. 1 depicts the SWAP platform that solves the problem offragmentation and provides a seamless connection between users fromseparate platforms. These two separate platforms are integrated by SWAP.Interactions between User 1 and User 2 are then sent for analysis. Theseinteractions are used in the Swap+ Profile, Elearning, and SWAP Project.

The derived information from analysis such as user emotion and mentalstates will be utilized in functioning of 3 major SWAP features—

-   -   1. Profiles (SWAP+)    -   2. Targeted Advertisement    -   3. Smart ELearning (addition to the chalkboard and virtual        classroom globe) SWAP+ Profiles

The way most social networking sites function, they mainly act as agreat platform for data storage, sharing and communication. But theythese are all a far cry from true social interaction simulation in otherwords in no way are these anywhere near how we interact in society. Thusthe profiles of SWAP+ will be a system which will be much closer to howwe remember people, conversations and moreover how we forget. The largeamount of data that get passed through the SWAP platform will beanalyzed and this data will be used to shape the SWAP+ profiles. The wayother people's SWAP+ profiles will appear to us. In this area we try tomimic the way in which we remember people. The profile's emotion feelwill be the general emotion that we generally exhibit when wecommunicate that with that person through any form of media (video, textor speech) (obtained from analyzed data from conversations takingplace). Keeping in trend with how we remember people in reality, sincehow a person is seen by is strongly shaped with event and experiences weshare with that person. The profile of the person will bear events,having strong emotions behind them. Any sort media—like text, speech,video or pictures. Texts can be presented simply as they are, videoswill we presented like snapshots with the option to be played by theuser. The SWAP+ profile can include:

-   -   1. Chatting threads (as depicted by FIG. 2)    -   2. Profile appearance (as depicted by FIG. 3)    -   3. Close friends (as depicted by FIG. 4)        1. Chatting Threads

The basic flaw which makes social interactions unrealistic is that everybit of data is remembered, unlike the case in real-life dailyinteractions. To replicate this communications that will be happeningthrough SWAP+ will follow a similar pattern. The comments of the threadwill slow start to deteriorate i.e. fade away. The period after whichthe part of the thread is completely forgotten will be a sort ofthreshold time, which will be close to average human being time formemory retaining. Memories having high cognitive strain or emotionattached will have much higher threshold time.

In FIG. 2, the comments of the thread will slow start to deterioratei.e. fade away. The period after which the part of the thread iscompletely forgotten will be a sort of threshold time, which will beclose to average human being time for memory retaining. Memories havinghigh cognitive strain or emotion attached will have much higherthreshold time. This example shows a conversation between two friendsdiscussing “last night's party.” The initial conversation contains lowemotionally attached or insignificant conversation (e.g. “Hey!” “Whathappened?” “Yeah?”). In the decayed conversation, however such aspectsof the conversation are decayed into visually smaller bubbles. Thelarger bubbles include phrases associated with high cognitive strain oremotion attached. In this example, one friend tells the other “I toldmom about how I lost her gold necklace.” This phrase is the largestbubbles, assuming that the friend was experiencing significantemotion—including perhaps anxiety, fear, etc.

2. Profile Appearance

In FIG. 3, the profile of SWAP+ will be dynamic in nature constantlychanging reflecting the mental state of the user. The real-time mentalstate will be determined from the various analysis methods, which willbe applied on the data passing through SWAP. Under a state of extremeemotion such as depression the user profile will be able to reflect thisstate. This will allow for other people to be notified of the user'semotional state and hence help him get back normalcy throughcommunication. Through analysis ‘close friends’ can also be identifiedwho under the above-mentioned situation will be notified. In thisexample, we see Abhishek Biswas' profile as seen by his girlfriend.Note: this profile is individualized. His girlfriend can see all theimportant conversations between them (as “decayed conversation”feature.) These conversations include highly emotional both positive andnegative phrases. Also, highly emotional paused scenes from videos willappear as well as pictures that have been discussed in emotionalconversations.

3. Close Friends

FIG. 4 demonstrates the analysis of data through SWAP for ‘closefriends’. The analysis will allow the application to identify peoplewith whom the user has discussions of high emotional content. A databaseof sorts can be created which will store people with whom the user hasdiscussion of high emotional content such as high positive emotioncontent, high negative emotion content, people with whom throughcommunication emotion changes from negative to positive. Also peoplewith whom there isn't communication of high emotion content but volumeand frequency of communication is very high, these people will also beidentified as ‘close friends’. When ever the user is in a state ofemotional extreme then the user's profile will be highlighted in thehomepages of the ‘close friends’. In this example, the friend whoseprofile shows high levels of distress is the largest. The user canvisually identify this friend and try to help her. The second largestpicture is also a friend who is visually distressed (which is seenthrough emotions detected on his profile) and is therefore seen as alarge image. The third largest image is the user's girlfriend's profile.Although her profile does not show high emotional context, her profileimage is highlighted because of the high volume and frequency ofcommunication.

Elearning

In virtual classroom or chalkboard feature the user may be required togo through leaning material or modules and solve problems. Based onobservation of Pupil dilation the cognitive load on user's mind can befound out. If the user is under high cognitive stress for prolongedperiod it is indicative that the user is unable to make progress withcurrent material or problem. Hence more comprehensive material may beprovided and in case problems a hint or a different problem may beprovided. Similarly the pupil study may also indicate the course andproblems may not cause appreciable cognitive strain so in this case acourse which is less comprehensive and problems of higher difficulty maybe presented. The SWAP feature will allow people from different videocommunication platforms to join into a virtual classroom. This virtualclass room will allow for multiple people to join at same time thecourse being taught will customized for each individual user. Thusstudent gets all the benefits of study in a classroom such discussion,debating, interactive doubt clearance, observing point of view of peers.At the same time the course is modified as peer the learning capacityand mental level of each individual student. So as the all students jointhe virtual classroom they all start out with the same course materialand as they carry forward with class, constantly each student cognitiveload level, attention, stress is being monitored. And based on this datamaterial is modified that will enable maximum learning will be provided.Apart from pupillary dilation and video analysis of face, eye trackingwill allowing monitoring the movement of the eyes hence it will bepossible to see whether that user is being able to focus on thematerial. Using eye tracking technology we can find the place where theuser is looking at and pattern recognition can be utilized to findwhether the material being presented is being read or not for exampleregularized movement of eyes indicate that the user is following thematerial presented and where as wandering and random movement of eyesare indicative that the material is not being followed.

The virtual classroom element of SWAP will have advanced tools tosimulate real class room like environment. The nature learning may be of2 types; video lecture and course material.

FIG. 5 shows the user interface for studying from video lecture. Thefollowing features are present: a notepad where the user can take roughnotes, inventory of study aids (like calculator), formula manual (forthe course being studied), and access to all rough notes. The work areacontains questions and problems that will be asked to the user as eachsub segment of the video lecture is completed for those of who finishthe problems quickly more problems will be asked and the next subsegment may start only after the minimum number of questions has beencompleted by everyone. The user will be able to communicate with hisother peers and ask them for help or doubt clearance (similar to realclass rooms). The feature will also be provided that allows for personwho is communicating with user to share his work sheet and help insolving and understanding the problem. As can be seen in this example,the lecture is synchronized with the notes in the formula manual beingdisplayed. Also, based on eye movement, pupillary dilation and facialstudy of other peers, the student (or teacher) can detect the amount ofdifficulty or ease his/her peers is having with the class and theproblems.

FIG. 6 shows a virtual classroom with e material used as learningmedium, based on study of eye movement, pupillary dilation and facialstudy. The material will be constantly modified and since all peers willbe present constant discussion will also be taking place.

If it is observed that the user wasn't taking in the course then pop upquestions will be presented on the work area, to check the usersunderstanding hence allow for optimised learning.

Also, based on eye movement, pupillary dilation and facial study ofother peers, the student can detect the amount of difficulty or easehis/her peers is having with the class and the problems. Areas that seemto be confusing for the student will be noted down and at the end ofeach study session these areas will be reviewed.

SWAP Projects

FIG. 7 shows an embodiment of a user interface. For example, anotherfeature that will be present along with the education globe is a singlesheet that can be shared across an entire group. All members of thegroup can make modifications to the sheet simultaneously. All editingand word processing features will be made available. This will allow forrapid completion of project with different parts (e.g. one person may bedrawing up some part while others may be writing) being done bydifferent people. In FIG. 7, for example, Linda, Jenny, and Robert canall see each others' videos. The “Toolbar” includes all possiblesoftware devices (e.g., tables, different languages, presentation tools,graphical tools, etc.) In this image, one of the users is able to seewhat he/she has entered into the project.

Since all progress being made is constantly visible to all the usersworking on it, a seamless integration will be possible. In factdifferent people can comment and suggest changes to some or more partsbeing done by someone else. Constant discussion and visibility amongstthe different team members will also be facilitated through audio andvideoconference, which will run in parallel with the SWAP Projectfeature. This will have huge utility in corporate sector, whichgenerally have members working on single project scattered all over theglobe.

FIG. 8 shows an embodiment of a collated version of the user interface.On a single screen, smaller images of various different types ofsoftware applications can be presented. Also each user's specific partis labelled automatically with their name. Thus, users are able to seethe different segments of the project that are being completed by otherusers.

Targeted Advertisements

Advertisement Will be Presented to Users Based on

-   -   a. Keyword matching    -   b. Based on knowledge of user's real-time emotional state.    -   c. Geographic location and movement pattern (for people using        mobile access medium like cell phones or tablets)

The advertisements that will be presented will be guided based on thecontent of the conversation, the mood of the user and the feature thatof SWAP that is being used.

For example people who show high level of cognitive stress may besuggested stress-relaxing medicine, names of specialists and people.People showing emotional extremes like extreme depression may besuggested holiday destinations and retreats, or books.

For mobile users the geographical location, path and movement pattern ofthe user will be taken into account to provide location based targetedadvertisement where product that might appeal to user (predicted bytaking into factors like nature of conversation, or media beingobserved, mood of the user and geographical position). This will enableadvertisement agencies to provide extremely specific advertisement.

Healthcare (Remote Diagnosis)

Advanced application can be developed which will collect data generatedfrom cell phones and transfer these to service provider who will analysethe data and transfer it to the healthcare agencies who can then providediagnosis on basis of the data provided.

Advancement in cloud computing enables us to utilise same apps fromdifferent computing devices like tablets, computers, laptops and cellphones. The apps thus developed will not device or platform specific butwill only be user specific, they will have an inventory of data miningmechanisms and depending on the device being used select mechanisms willbe used.

Combination of data collected from the multiple sources will used todetermine lifestyle of the person and this can be used by healthcare andinsurance industries. This cycle is depicted in FIG. 9, which shows anembodiment of how insurance or health care companies will acquire datethrough the cell phone of a client. The cellular data will go through atelecom operator, through a 3^(rd) party service provider, who willcollect and analyze the data and return it back to the company.

3rd party provider can collect this data only after approval from theindividual who owns the cellular device over a set period of time. Thedata can be used by the individual for personal usage or along withhis/her doctor for health analysis. For example, an individual who isfighting with obesity can have his/her cellular data tracked for onemonth. After analysis of this data, the doctor and patient (e.g., anobese individual) can work together to target some of the problems thatthe patient. On the other hand, health insurance companies can use thisdata after approval from the potential customer to determine howhealthily he/she is living. If the behavioural choices, emotions, andother everyday actions of the customer seem to promote healthylifestyle, the insurance company can give discounted rates to such acostumer. There are three methods by which current day smart phones candetermine the lifestyle, behaviour, or emotions of a person. Time andlocation, the audio vector of the cellular device, and typingcharacteristics can be used to analyse a person's health. This data willbe collected over a span of time.

Lifestyle Data Will Include:

1. Location information

2. Driving information and movement information

3. His affective state and average cognitive state

4. Habitual information—diet, drinking, etc.

5. Real-time information about physical health

The span of time and monitoring parameters will be determined jointly byuser and concerned agency.

1. Location Information:

The geographical location of a person can give a general idea of theperson's life style and personality. Information like movement overdifferent non-urban terrain is indicative of an adventurous lifestyle.Also information like the places the person visits will highlight manyof the persons traits e.g., location data showing that one visitsMcDonald's everyday indicates that the individual does not have ahealthy lifestyle, compared to an individual who visits the gym on adaily basis. After large enough samples of data are collected, amovement map of the individual can be created that shows frequencies ofvisits to certain locations within a certain range. Using a patternidentification algorithm, doctors or life insurance agencies can moreeasily analyse location data of an individual and correlate this tohis/her lifestyle.

2. Driving Information and Movement Information:

Velocity and acceleration analysis can be done by the GPS on the phoneto determine whether or not the individual is a rash driver. Informationabout speed limits on a majority of roads is present on the maps thatare on smart phones. It can be understood that an individual is drivingif they are on a road that is traversed upon by vehicles. Usually, GPStracking provides an accuracy of within 50 meters. So, the speed of aperson can be determined by dividing each 50-meter path covered by thetime required by the individual to traverse that distance. It will benoted that a person is walking, not driving, on such a road if the speedis significantly below that of the speed limit (like below 10 km/s) foran extended period of time. Even this information is vital, as itinforms that the individual is walking on a road that is meant forvehicles, which in itself is an unsafe behaviour. This behaviour willnot be confused with cars that are just stuck in traffic, becausetraffic patterns are now being updated constantly to smart phones, anddata about the location and time of the traffic can easily be collected.After confirming that the individual is driving on the road, one cancompare the speed of his/her vehicle with the speed to determine whetheror not the person is speeding. Even if the individual whose data isbeing taken down is not the driver, it is important to know if theindividual is at risk by being in the same vehicle as a person who isspeeding. In addition, if the average velocities recorded in each 50meter block are fluctuating highly, and the time taken to cover one 50meter stretch is significantly different than the time taken to coveranother, one can see that the driving is “stopping and going” toofrequently. An accumulation of data about velocity can easily betranslated into acceleration analysis, where the rashness of the driverwith sudden accelerations can be determined.

3. The Affective and Cognitive State:

The user emotional and cognitive data will obtained from allcommunications taking place in form texting, video chat and audio chatfrom devices like smart phones, tablets computers or laptops. Since thefunctioning of various features of SWAP like profile+ and virtualclassrooms is heavily of dependent on user emotion and cognitive statethe apps can gather data from these features to observe emotional andcognitive states of the user during the period of observation. Thesedata can be combined with location data (considering the fact that theuser is constantly carrying his smart phone) to affect map of theperson. The affect map will show which emotions and mental statecorrespond to specific locations of the individual.

4. Habitual Information:

Various apps and detection mechanisms can be utilised to determinevarious habits of the user like eating habits, drinking habit, smokinghabit, etc. Apps like MEALSAPP®, etc. can be detected by the advancedapps of SWAP and used to detect traits of the user.

5. Physical Health Information:

Smart phones have pedometers installed in them and also have thecapacity to find a person's pulse. All these features can be used byadvanced SWAP apps to give a person's physical health status which canbe further combined with time and location information supplement theabove mentioned data.

From this network, an emotional map can also be constructed that showswhich emotions correspond to specific locations of the individual. Thislocation tracking combined with the audio vector and typing analysis canindicate which locations the individual should continue going to boosthappiness and which locations should be avoided, as they may becorrelated to stress, anger, sorrow, etc.

Emotion Analysis

The large amount of data that will be passing through SWAP will beanalysed in following ways:

1. Video Analysis

2. Speech Analysis

3. Typing Analysis

FIG. 10 illustrates an embodiment of how the media acquired by SWAP isgoing to be analyzed. The data can be organized into three segments:Video, Text, and Audio. Pupillary dilation analysis and facial featureanalysis can be taken from the video data analysis. From textual data,keywords, knowledge-based artificial neural networks, typing speed,pressure, contextual clue and error analyses can be done. From audiodata, features can be extracted and analyzed. These can be used todetermine emotion.

Video Analysis

a. Facial Emotion Recognition

-   -   The emotion of user is recognized by tracking the movements of        some fixed points of the face like the corners of eyes, mouth        boundary, etc. The amount of movement of these points in various        frames of the video are constantly monitored and the data thus        generated is fed in various classifiers like Bayesian Networks,        Decision Trees etc. from which we find the emotion of the user.

b. Pupillary Dilation

-   -   Dilation of pupils is common phenomena. The causes for dilation        of pupils are:        -   1. Mental stress (cognitive load).        -   2. Emotion        -   3. Light stimulus

Our pupils tend to dilate in different emotional situation. Studiesconducted have shown that with increase in arousal level the diameter ofout pupils increase. Also valance causes our pupils to dilate. But theamount of dilation caused for positive and negative emotion has beenfound out to be the same. This issue may be resolved with further studyin this area—analyzing the rate of dilation and dilation period and alsothe amount and rate of dilation under combination of different stimuli.Also while measuring pupil dilation, the dilation caused due otherstimuli like light have 2 either ignored or factored out (more study isrequired in this area). Pupillary dilation is a complete involuntaryreflex and hence there no change for us to consciously control it. (Thisis possible in case facial emotion recognition.) Hence no emotion fakingis possible. A distinct difference is apparent for male and femaleusers. So gender classification can be done easily through study ofpupil dilation pattern.

2. Speech Analysis

To find out emotion from speech the basic idea is to study the way thevoice box functions while producing speech under different emotionalstates. Depending upon how it functions variations in wave form appear.By extracting the various features of the waveform from which thesevariations can be detected and putting these (certain combinations offeatures) into various soft computing models the emotion can bepredicted.

Data extracted from an audio vector can be used to determine one'semotional state. The volume and pitch of the speaker can be foundwithout actually recording what the speaker is saying, avoiding anyinvasion of privacy. The content of the conversation is immaterial tothe 3^(rd) parties, since only the tonal nature (loudness and frequency)of the individual is being analyzed.

To find emotion from speech first we extract various components ofspeech, which carry data with respect to emotion. These components areenergy, pitch, cross sectional area of vocal tract tube, formant, speechrate and spectrum features and spectral features like linear predictioncoefficients (LPC), linear prediction cepstrum coefficients (LPCC), Melfrequency cepstrum coefficients (MFCCs) and its first derivative andlog-frequency power coefficients (LFPCs). All these components areextracted from the original speech waveform using various mathematicaland statistical techniques. The features can be extracted utilizingvarious combinations of the features. These acoustic features are usedto find out emotions through various classifiers.

Methods that classify emotions from prosody contours are neuralnetworks, multi-channel hidden Markov model, mixture of hidden Markovmodels these give prediction from the temporal information of speech

Methods which classify emotions from statics of prosody contours supportvector machines, k-nearest neighbours, B ayes classifiers using pdf(probability distribution functions) generated by Parzen windows, Bayesclassifier using one Gaussian pdf, Bayes classifier using mixture ofGaussian pdfs.

Hence from the above mentioned soft computing techniques we find theemotion of a person. From this his type of collection over a large spanof time, general emotional status can be determined via the audiovector.

Data extracted from an audio vector can be used to determine one'semotional state. The volume and pitch of the speaker can be foundwithout actually recording what the speaker is saying, avoiding anyinvasion of privacy. The content of the conversation is immaterial tothe 3^(rd) parties, since only the tonal nature (loudness and frequency)of the individual is being analysed.

Typing Analysis

We will utilize the following methods to kind find emotion of the userfrom the text that he types. All the methods will be working inparallel.

-   -   1. Finding emotional keywords in textual data and deriving the        emotion of the sentence from that.    -   2. Finding emotion from sentences, lacking emotion key words        using Knowledge Based Artificial Neural Networks.    -   3. By analyzing the typing speed. The various features of typing        that we study are time lag between consecutive keystrokes    -   4. Error level. (Number of times corrections are made in the        sentences).    -   5. Pressure Analysis—the pressure sequence various features        extracted like mean, standard deviation, maximum and minimum        energy difference, the positive energy center (PEC) and the        negative energy center (NEC). PEC and NEC are calculated from        mean and standard deviation after normalization).    -   6. Contextual cue analysis weather, lighting, temperature,        humidity, noise level and shaking of the phone

The various features of typing that we study are time lag betweenconsecutive keystrokes, number of times back space is used, typing speedand pressure put behind each keystroke, for example, from the pressuresequence various features extracted like mean, standard deviation,maximum and minimum energy difference, the positive energy centre (PEC)and the negative energy centre (NEC). PEC and NEC are calculated frommean and standard deviation after normalisation). Apart from thesevarious contextual cues are also taken into account like weather,lighting, temperature, humidity, noise level and shaking of the phone,and the frequency of certain characters, words, or expressions can beused to determine emotion. The above mentioned sets of features are fedinto various soft computing models (like support vector machines,Artificial neural networks, Bayesian networks, etc), these generateprobability towards a particular emotional state individually for eachset of features. Also since in most cases the outcome will be towardsthe same emotion from computations on each feature set hence fusionmethods can be used to compute the over all probability of having thatparticular emotion by combining the individual results.

Towards Development of a Model for Emotion Detection from TypingAnalysis

First we find out features of typing which is exhibited by most peopleand features of these patterns which detect emotions. We now developvarious soft computing models which allow for the detection of aparticular emotion from the typing pattern. To see the efficiency andfunctionality of these models we conduct sample studies where a softwareis downloaded by the people who's typing pattern will be analysed. Apartfrom the typing pattern detection another detection method will also bethere to measure the emotional state at the time of typing. These 2methods will work in parallel and the emotion detected by latter methodwill be taken as reference and later during analysis it will be seenwhether the emotion predicted by the former method matches with thereference.

In the latter method the peoples' emotional valence will be detected bystudy of their facial muscles which can be done by use of a simpleweb-cam (generally available with their computer or laptop) and arousalwill be detected by measuring the galvanic conductivity of skin measuredwith wristband with this capability (already a commercial productmanufactured by a company called AFFECTIVA®).

The above mentioned method departs away from way experiments have beendone on typing analysis recently. In these experiments the candidateswho's pattern will be analysed are given the software which analyses thetyping pattern but reference emotion is found out through questionnairesthat enquire about the emotion of the person before he starts to type.

Again, this will not be a privacy issue because these third parties willnot access full texts. They will just automatically search through themfor the frequency of specific words or expressions that may correlate tothe individual's emotions. These data will not just be collected once,but over a long span of time. As a result, the overall emotional andbehavioural state of individual will be determined. So, a person typingvery fast on a shaking phone, with high pressure under the keys, andusing a high frequency of unpleasant words used in his/her texts canreveal anger or stress. However, if data that points to this behaviouris only collected once or twice in a span of a month, it will not beregarded as very important, as everyone has some infrequent expressionsof anger or stress. However, if a majority of typing data is like this,a doctor of insure company can infer that the individual is constantlyangry or stressed out, which is not good for health.

Mental Health Tracker

Currently 1 in 4 Americans have a mental disorder. It is becomingincreasingly important to identify mental disorders at younger age, whensymptoms are still slight. It is thus essential for primary carephysicians in addition to psychiatrists to be able to recognize mentaldisorders.

In an embodiment, the DSM IV-TR (Diagnostic and Statistical Manual forMental Disorders) and DSM IV-PC (Diagnostic and Statistical Manual forPrimary Care) version, which are the manuals used by doctors todetermine both the presence and category of mental disorder, could beincluded in as part of a computerized algorithm to help doctors forpatient tracking. The DSM IV-PC (meant for primary care physicians, whoare not specialized in mental disorders) has organized symptoms thatcreate a diagnostic algorithm. This manual is concise and fullycompatible with the wider used DSM IV-TR, which is used by psychiatrics.

Primary care physicians (PCP) have made many initial diagnoses of mentaldisorders. However, many diagnoses remain undetected, as PCPs generallyonly have check-ups with patients one or twice a year, and mentaldisorders, at first may be difficult to observe, as there are nostandardized tests for mental disorders. Due to the difficulty indiagnosing a mental disorder within the limited face-to-facepatient-doctor interaction, it can be extremely helpful for doctors touse a non-invasive patient tracking method of an embodiment as shown inFIG. 11, which shows the main aspects of SWAP that can be used to createa profile of the patient that can then be analyzed by the algorithm ofthe DSM-IV and by doctors.

Doctors can track their patients using methods detailed in otherexamples of our patent. FIG. 12 shows a flow diagram that delineatespossible tracking mechanisms. FIG. 12 shows that the proposed trackercan use video data, time and location analysis, typing analysis, andaudio data in order to understand the patient's emotional state. Over aweek or month long analysis, this tracker will then use an algorithmfrom the DSM-IV in order to identify an initial mental diagnosis. Withthe use of the guidelines in the DSM-IV-PC, the algorithms created bythe manual can be used along with our tracking system to provide aprimary initial screening for patients for detection and type of mentaldisorder. Thus, SWAP's Mental Health Tracker can help a physician betterunderstand his patient's needs.

APPENDIX 1

The U.S. patents and publications listed below are hereby incorporatedherein by reference in their entirety. U.S. Pat. No. 8,102,406; Issuedate: Jan. 24, 2102; Method and system for producing a video synopsisU.S. Pat. No. 8,073,839; Issue date: Dec. 6, 2011; System and method ofpeer to peer searching, sharing, social networking and communication inone or more networks U.S. Pat. No. 7,523,163; Issue date: Apr. 21, 2009;Distributed network system architecture for collaborative computing U.S.Pat. No. 7,313,595; Issue date: Dec. 25, 2007; System and method forrecord and playback of collaborative web browsing session U.S. Pat. No.7,236,926; Issue date: Jun. 26, 2007; System and method for voicetransmission over network protocols U.S. Pat. No. 6,567,813; Issue date:May 20, 2003; Quality of service maintenance for distributedcollaborative computing Publication number: US 2011/0258125; Filingdate: Apr. 14, 2011; Collaborative social event planning and executionPublication number: US 2011/0225519; Filing date: Feb. 16, 2011 Socialmedia platform for simulating a live experience Publication number: US2011/0066664; Filing date: Sep. 15, 2010; Sports collaboration andcommunication platform Publication number: US 2010/0299334; Filing date:Sep. 8, 2009; Computer implemented system and method for providing acommunity and collaboration platform around knowledge transfer,expertise, innovation, tangible assets, intangible assets andinformation assets Publication number: US 2010/0332616; Filing date:Aug. 31, 2009; Web guide Publication number: US 2010/0262550; Filingdate: Apr. 8, 2009; Inter-corporate collaboration overlay solution forprofessional social networks Publication number: US 2009/0094039; Filingdate: Oct. 4, 2007; Collaborative production of rich media contentPublication number: US 2008/0297588; Filing date: May 31, 2007, Managingscene transitions for video communication Publication number: US2005/0198141; Filing date: Feb. 4, 2005; Secure communications systemfor collaborative computing Publication number: US 2003/0167304; Filingdate: Dec. 29, 2000; Distributed meeting management Publication number:US 2003/0164853; Filing date: Dec. 29, 2000; Distributed documentsharing

What is claimed is:
 1. A method of establishing a classroom comprising:teaching a course having a course content that is taught online to aplurality of members through a video-lecture using a web-cam, whereinthe course requires each of the plurality of the members to go through acourse content, obtaining a video input from each of the plurality ofmembers, wherein the video input comprises video images obtained fromthe web-cam, the video images comprising images of pupil dilation, eyetracking and/or facial features of each of the plurality of the members,analyzing in real-time the video images using a video analysis system,performing a collaborative interactive session among the plurality ofmembers, determining, using the video analysis system, a learningcapacity or mental stress level of each of the plurality of membersbased on observation of the images of pupil dilation, eye trackingand/or facial features, modifying the course content presented to eachof the plurality of the members as per the learning capacity or mentalstress level of each of the plurality of members, presenting questionsto each of the plurality of the members during a sub-segment of thevideo lecture, and starting a next sub-segment of the video lecture onlyafter a minimum number of the questions are completed by all of theplurality of the members.
 2. The method of claim 1, further comprisingdisplaying of targeted advertisements or notifications based on thecontext of the interactive collaborative session.
 3. The method of claim2, further comprising measuring effectiveness of the displaying oftargeted advertisements or notifications.
 4. The method of claim 1,further comprising integrating an application or a device within thecollaborative interactive session.
 5. A computer implemented systemcomprising: a course having a course content that is taught online to aplurality of members through a video-lecture, a web-cam, a video inputfrom each of the plurality of members, video images comprising images ofpupil dilation, eye tracking and/or facial features of each of theplurality of the members, a video analysis system that analyzes inreal-time the video images, and a work area comprising questionspresented to each of the plurality of the members during a sub-segmentof the video lecture; wherein the computer implemented system isconfigured to perform a collaborative interactive session among theplurality of members and obtain the video input from each of theplurality of members, wherein the video analysis system is configured,based on observation of the video images of the pupil dilation, eyetracking and/or facial features, to determine a learning capacity ormental stress level of each of the plurality of members, and wherein thecomputer implemented system is configured to modify the course contentpresented to each of the plurality of the members as per the learningcapacity or mental stress level of each of the plurality of members andpermit a next sub-segment of the video lecture to start only after aminimum number of the questions from the work area are completed by allof the plurality of the members.
 6. The system of claim 5, wherein thecourse content comprises learning materials or modules comprising videolectures and course materials.
 7. The system of claim 5, furthercomprising an user interface for studying from a video lecture, the userinterface comprising a notepad and an inventory of study aids.
 8. Thesystem of claim 5, wherein the learning capacity or mental stress levelcomprises a level of difficulty or ease that each of the plurality ofthe members is having in classroom.
 9. The system of claim 5, whereinall the plurality of the members join the classroom with a same coursecontent and the plurality of the members move forward in the classroom,the course content for each of the plurality of the members is modified.10. The system of claim 5, wherein at least some of the plurality ofmembers interact from different human interaction platforms, wherein thedifferent human interactions platforms comprise social media platforms.11. The system of claim 5, wherein the system is further configured todisplay targeted advertisements or notifications based on the context ofthe interactive collaborative sessions.
 12. The system of claim 11,wherein the system is further configured to measure effectiveness of thedisplaying of targeted advertisements or notifications.
 13. The systemof claim 5, wherein the system is further configured to integrate anapplication or a device within the collaborative interactive session.14. The system of claim 5, wherein the system comprises a sound and/orvideo hub, wherein the sound and/or video hub allows any member of theplurality of the members to play a song and/or a video andsimultaneously allows the at least one of the plurality of members tolisten and/or watch the song and/or the video played.
 15. The system ofclaim 5, wherein the system comprises audio and/or video synopsis of thecollaborative interactive session for the plurality of members using asound and image-processing technology that creates a summary of anoriginal full length audio and/or video.
 16. The system of claim 5,wherein the system is configured to determine a mental health of the atleast one of the plurality of members by analyzing audio, video, textualand location data of the at least one of the plurality of members, andevaluating the data in a standardized model.
 17. A tangiblenon-transitory computer readable medium comprising computer executableinstructions, that when executable by one or more processors, cause acomputer implemented system to conduct the method of claim
 1. 18. Themedium of claim 17, further comprising computer executable instructionsexecutable by one or more processors for displaying of targetedadvertisements or notifications based on the context of the interactivecollaborative sessions.
 19. The medium claim 17, wherein the executableinstructions comprise instruction for determining a mental health of theat least one of the plurality of members by analyzing audio, video,textual and location data of the at least one of the plurality ofmembers, and evaluating the data in a standardized model.